Repository navigation
Add auto-review admonishments and instructions to try and improve generated code - #5103
Conversation
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
…build Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
An unattended session no longer waits on a permission prompt. The agent can send unreviewed code by prefixing the command with VG_SEND_UNREVIEWED=1. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
|
This seems to be OK at finding potential bugs (among a bunch of non-bugs for cases that implicitly aren't allowed, which would be useful if we were trying to program that defensively). It doesn't really seem to think about design or catch bad design; it really really likes tests and comprehensiveness and error handling. It might be missing a "design lens". |
The lens draws on vg review comments from the last two months, Real Python's checklist for AI-generated code, and screened web research. Tests may share code when each test still reads as plain statements. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
|
I asked for a notion of design review, using the Semble vector database to try and enable the review process to detect duplicate ideas across the codebase without trying to load the whole project up into a language model context. I have no idea whether this works well, since the circular evaluations used during generation can't really count, but we will find out. |
|
There are some new generated TODOs in the surjector that came from comments from the test reviews. I think they indicate places where we might want reassuring comments more than places where we really need behavior to change, but I asked for them to be added so they could maybe eventually be answered. |
Changelog Entry
To be copied to the draft changelog by merger:
Description
This adds some skill files adapted from those shared with me by Maximilian Haeussler from the Genome Browser group, which they are using for their own automated post-merge code review.
I sent the robot to try and do some example reviews and compare them against actual reviews of those PRs, and to then try and tweak the instructions to improve the automated reviews. "Just tell system to work better" is not ranked very high on the list of validated optimization techniques, so it's not clear if that approach will have helped. If you are getting reviews from this you do not like, please double down on this dubious technique and keep admonishing it to improve.